An easy change we can split out of elastic#121885 to make that shorter.

…astic#122188)

An easy change we can split out of elastic#121885 to make that shorter.

…22188) (#126293)

An easy change we can split out of #121885 to make that shorter.

…121885)

This change moves the query phase a single roundtrip per node just like can_match or field_caps work already.
A a result of executing multiple shard queries from a single request we can also partially reduce each node's query results on the data node side before responding to the coordinating node.

As a result this change significantly reduces the impact of network latencies on the end-to-end query performance, reduces the amount of work done (memory and cpu) on the coordinating node and the network traffic by factors of up to the number of shards per data node!

Benchmarking shows up to orders of magnitude improvements in heap and network traffic dimensions in querying across a larger number of shards.

…#126563)

* Introduce batched query execution and data-node side reduce (#121885)

This change moves the query phase a single roundtrip per node just like can_match or field_caps work already.
A a result of executing multiple shard queries from a single request we can also partially reduce each node's query results on the data node side before responding to the coordinating node.

As a result this change significantly reduces the impact of network latencies on the end-to-end query performance, reduces the amount of work done (memory and cpu) on the coordinating node and the network traffic by factors of up to the number of shards per data node!

Benchmarking shows up to orders of magnitude improvements in heap and network traffic dimensions in querying across a larger number of shards.

* Filter out empty top docs results before merging (#126385)

`Lucene.EMPTY_TOP_DOCS` to identify empty to docs results. These were previously
null results, but did not need to be send over transport as incremental reduction
was performed only on the data node.

Now it can happen that the coord node received a merge result with empty top docs,
which has nothing interesting for merging, but that can lead to an exception because
the type of the empty array does not match the type of other shards results, for
instance if the query was sorted by field. To resolve this, we filter out empty
top docs results before merging.

Closes #126118

---------

Co-authored-by: Luca Cavanna <[email protected]>